Automatic image analysis and quantification for fluorescence in situ hybridization

ABSTRACT

An analysis system automatically analyzes and counts fluorescence signals present in biopsy tissue marked using Fluorescence in situ Hybridization (FISH). The user of the system specifies classes of a class network and process steps of a process hierarchy. Then pixel values in image slices of biopsy tissue are acquired in three dimensions. A computer-implemented network structure is generated by linking pixel values to objects of a data network according to the class network and process hierarchy. Objects associated with pixel values at different depths of the biopsy tissue are used to determine the number, volume and distance between cell components. In one application, fluorescence signals that mark Her2/neural genes and centromeres of chromosome seventeen are counted to diagnose breast cancer. Her2/neural genes that overlap one another or that are covered by centromeres can be accurately counted. Signal artifacts that do not mark genes can be identified by their excessive volume.

TECHNICAL FIELD

The present invention relates generally to locating specified imagestructures in digital images, and more specifically to acomputer-implemented system for automatically identifying andquantifying cellular structures marked using fluorescence in situhybridization (FISH).

CROSS REFERENCE TO COMPACT DISC APPENDIX

The Compact Disc Appendix, which is a part of the present disclosure, isone recordable Compact Disc (CD-R) containing information that is partof the disclosure of the present patent document. A portion of thedisclosure of this patent document contains material that is subject tocopyright protection. All the material on the Compact Disc is herebyexpressly incorporated by reference into the present application. Thecopyright owner of that material has no objection to the facsimilereproduction by anyone of the patent document or the patent disclosure,as it appears in the Patent and Trademark Office patent files orrecords, but otherwise reserves all copyright rights.

BACKGROUND

Systems for detecting and analyzing target patterns in digital imageryhave a wide variety of uses. One such use is analyzing anatomicalregions in radiological images. For example, systems for analyzingcomputed tomography (CT) images are used for the computer-aideddetection of cancerous regions in human breasts and lungs. Another usefor such image analysis systems is to detect and analyze target patternsin biomedical images obtained from microscopes, such as confocalmicroscopes. For example, pathologists use confocal microscopes toanalyze cells and their components, such as organelles, membranes,nuclei, genes, chromosomes and macromolecules such as RNA, DNA, proteinsand peptides. Such image analysis is used not only in diagnosis andprognosis relating to medical patients, but also in basic research, drugdiscovery and clinical trials.

Confocal microscopy offers several advantages over conventional opticalmicroscopy by providing a wider depth of field, eliminating out-of-focusglare, and allowing the collection of serial optical sections from thickspecimens. The laser scanning confocal microscope (LSCM) is currentlythe most widely used confocal microscope for biomedical researchapplications. In the biomedical sciences, a major application ofconfocal microscopy involves imaging cells and cell components that havebeen labeled with biomarkers, such as fluorescent probes. Confocalmicroscopy in combination with in situ hybridization and fluorescencetechniques can be used to study DNA and RNA sequences in chromosomes andto analyze cell components, such as chromosomes and genes. One suchtechnique for analyzing cell components is Fluorescence in situHybridization (FISH). For additional information on the FISH technique,see U.S. patent application Ser. No. 11/050,035, published on Dec. 1,2005 as Publication No. 2005/0265588 A1, by Gholap et al. (the entiretyof which is incorporated herein by reference).

In one specific application, FISH is used to analyze the Her-2/neuralgene in breast biopsies in order to provide a diagnosis and prognosisfor breast cancer. Using confocal microscopy in combination with FISH, apathologist calculates the degree of gene amplification as the basis forthe diagnosis. In one accepted diagnostic procedure, the pathologistanalyzes a minimum of one hundred nuclei in order to calculate thedegree of amplification. In this conventional procedure, the pathologistmanually counts the marked chromosomes and genes (called “fluorescencesignals”) in each of the one hundred nuclei and then calculates theratios of the genes to the chromosomes. A disadvantage of thisconventional procedure is that even an experienced pathologist may misssome of the fluorescence signals due to fatigue and loss ofconcentration. Most of the biopsies contain normal counts of markedgenes and chromosomes, and the pathologist may lose concentration withthe tedium of counting genes and chromosomes in hundreds of nuclei inmultiple biopsies. Moreover, determining whether a fluorescence signalrepresents a single gene or multiple overlapping genes based on thebrightness and size of the fluorescence signal is often a subjectivedetermination. Individual pathologists may have different countingstyles.

Thus, an automated system for counting fluorescence signals obtainedfrom the FISH technique is desired. Existing automated counting systemscount fluorescence signals based on two-dimensional images. See, e.g.,Gholap et al., Pub. No. 2005/0265588 A1, cited above. Even in existingsystems that obtain three-dimensional information using confocalmicroscopy, however, the systems analyze two-dimensional images obtainedby condensing the three-dimensional information, thereby losing much ofthe three-dimensional information. It is difficult to distinguishindividual nuclei and other cell components in two-dimensional compositeimages obtained by condensing three-dimensional information.Fluorescence signals that touch or overlap other signals cannot beaccurately counted. In addition, information concerning the distancebetween signals and the size of individual signals is lost. Thus, asystem is sought for automatically counting fluorescence signals thatare present in three dimensions in slides obtained using the FISHtechnique.

SUMMARY

An image analysis system extracts, segments, classifies, quantifies andcounts three dimensional objects present in tissue, such as biopsytissue of a breast cancer patient. The analysis system automaticallyanalyzes and counts fluorescence signals present in the biopsy tissuemarked using the Fluorescence in situ Hybridization (FISH) technique.The user of the system specifies classes of a class network and processsteps of a process hierarchy. Then pixel values in image slices of thebiopsy tissue are acquired in three dimensions. Each separate imageslice of a z-stack is acquired at a different depth in the z-dimensionof the biopsy tissue. A computer-implemented network structure isgenerated by linking pixel values of the image slices to objects of adata network according to the membership functions of the class networkand the process steps of the process hierarchy.

Objects associated with pixel values at different depths of the biopsytissue are used to determine the number, volume and distance betweencell components. For example, the distance between genes and between agene and the nuclear membrane can be determined. In one application,fluorescence signals that mark Her-2/neural genes and centromeres ofchromosome seventeen are counted to obtain a diagnosis of breast cancer.Her-2/neural genes that overlap one another or that are covered bycentromeres can be accurately counted. Signal artifacts that do notdesignate genes can be identified by their abnormal area or volume.

A method of automatically counting cellular components enhances thereliability and confidence of the FISH diagnostic technique. The methodeliminates the human error associated with conventional manualdiagnostic procedures using the FISH technique. Moreover, the method ismore accurate than existing automated counting methods based ontwo-dimensional image analysis. The method enables the user quickly andeasily to quantify the cellular components in hundreds of nuclei, inmultiple regions of a slide, and in multiple slides of a biopsy. Inaddition, the method enables the analysis of slides from multiplebiopsies. Consequently, the method delivers meaningful diagnosticsupport based on multiple biopsies, which cannot be provided by existingautomated counting methods based on two-dimensional image analysis.

The method can also analyze images captured in multiple focal planes atmultiple wavelengths using multiple biomarking techniques and spectralmethods. For example, multiple images of the same biopsy tissue takenusing a microscope, an X-ray device, a computer tomograph, an ultrasoundimaging device, and a magnetic resonance imaging device can be analyzedand correlated. In addition, the same types of cell components in thevarious images can be labeled using different biomarkers. Objects thatcorrespond to the same cell components in the various images are thenlinked, correlated, analyzed and counted. The diagnosis and prognosis ofthe patient is improved by correlating the results of the analysis ofthe various images taken using the different biomarkers and spectralanalysis techniques.

In one specific embodiment, a software program receives a specificationof a class network and a specification of a process hierarchy. Pixelvalues of an image that includes cell components marked usingfluorescence in situ hybridization (FISH) are acquired. The softwareprogram then performs the process steps of the process hierarchy togenerate a data network by linking selected pixel values of the image toobjects. The objects are then classified according to the membershipfunctions of each class and subclass of the class network. One of theclasses corresponds to a particular type of marked cell component. Thesoftware program then counts the number of the particular type of markedcell component using the data network.

Other embodiments and advantages are described in the detaileddescription below. This summary does not purport to define theinvention. The invention is defined by the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, where like numerals indicate like components,illustrate embodiments of the invention.

FIG. 1 is a diagram illustrating the process of analyzing nuclei frombiopsy tissue of a human breast.

FIG. 2 shows a slide containing the biopsy tissue of FIG. 1.

FIG. 3 is a diagram of multiple image slices taken from the slide ofFIG. 2.

FIG. 4 is a diagram showing cell components in a nucleus from an imageslice of FIG. 3.

FIG. 5 is a three-dimensional diagram of cell components that are to becounted.

FIG. 6 is a two-dimensional composite diagram of the multiple imageslices of the three-dimensional diagram of FIG. 5.

FIGS. 7A-C show actual microscopic images of nuclei, genes andcentromeres that are counted by the analysis system according to theinvention.

FIG. 8 is a microscopic images of nuclei containing green fluorescencesignals that mark the centromeres of chromosome number seventeen andpink Her-2/neu fluorescence signals.

FIG. 9 is a schematic diagram representing part of a data network basedon pixel values from the three image slices of FIG. 5 that cut throughthe two nuclei.

FIG. 10 is a simplified schematic diagram of a computer-implementednetwork structure that includes a data network, a class network and aprocess hierarchy.

FIG. 11 shows digital image slices at various depths of a slidecontaining biopsy tissue.

FIG. 12 shows the red, green and blue image slice components that makeup the image slices of FIG. 11.

FIG. 13 is a flowchart of steps for analyzing and counting cellcomponents using the computer-implemented network structure of FIG. 10.

FIG. 14 is a diagram showing the class network of FIG. 10 in moredetail.

FIG. 15 is a flowchart showing the substeps of the first step of FIG. 13for specifying the class network of FIG. 10.

FIG. 16 is a diagram showing the process hierarchy of FIG. 10 in moredetail.

FIG. 17 is a screenshot of the graphical user interface of the analysissystem containing an outline representation of the process hierarchy ofFIG. 16.

FIG. 18 is a flowchart showing the substeps of the second step of FIG.13 for specifying the process hierarchy of FIG. 16.

FIG. 19 is a screenshot of a pop-up window generated by the analysissystem to assist in specifying an algorithm in a step of FIG. 18.

FIG. 20 is a diagram showing representations of various types of linksin the computer-implemented network structure of FIG. 10.

FIG. 21 is a simplified schematic diagram of the computer-implementednetwork structure of FIG. 10 after the class network and processhierarchy have been specified in the specification mode.

FIG. 22 is a simplified schematic diagram of the computer-implementednetwork structure of FIG. 10 after the data network has been generatedin the execution mode.

FIG. 23 is a simplified diagram of a data network in which pixel valuesare linked to objects.

FIG. 24 is a simplified diagram of a data network in which objects arelinked to other objects by various types of links.

FIG. 25 is a flowchart showing the substeps of the fifth step of FIG. 13for generating the data network of FIG. 10.

FIG. 26 is a screenshot of a graphical user interface generated by theanalysis system to assist in the editing of the class network andprocess hierarchy of FIG. 10.

FIG. 27 is a screenshot of the graphical user interface generated of theanalysis system with a window showing an outlined Her-2/neu gene in anucleus.

FIG. 28 is a screenshot of the graphical user interface generated of theanalysis system with a window showing the outline of what appears to bean Her-2/neu gene, but is actually an imaging artifact.

FIG. 29 is a flowchart showing the substeps of a step of FIG. 25 forexecuting the process steps of the process hierarchy of FIG. 10.

FIG. 30 is a flowchart showing the substeps of a step of FIG. 29 forgenerating a domain specified in a process step.

FIG. 31 is a simplified schematic diagram of a cognition network whenthe data network has been generated from many digital images, such asfrom many image slices of biopsy tissue.

FIG. 32 is a screenshot of a process hierarchy used by anotherembodiment of the computer-implemented network structure of FIG. 10 toanalyze individual cells in a cell assay.

FIG. 33 is a 3-dimensional diagram of a dividing cell output in the laststep of FIG. 13 with marked target objects.

FIG. 34 is a listing of high-level lines of XML code corresponding to aCognition Language script that implements a class network and a processhierarchy for analyzing and counting fluorescence signals present inbiopsy tissue marked using the FISH technique.

FIGS. 35A-E show more lines of the XML code of FIG. 34.

DETAILED DESCRIPTION

Reference will now be made in detail to some embodiments of theinvention, examples of which are illustrated in the accompanyingdrawings.

An analysis system is disclosed that automatically counts fluorescencesignals present in biopsy tissue marked using the FISH technique. FIG. 1illustrates the process of analyzing nuclei from biopsy tissue 10 takenfrom a human breast 11. First, the biopsy tissue 10 is extracted fromone location in the breast 11 of the patient. Only one or just a fewbiopsy samples are taken partly because the biopsy procedure is painful.Taking many biopsy samples also damages the structure of the breast andis also typically avoided for both medical and aesthetic reasons. Underconventional analysis methods, only about one hundred nuclei of thebiopsy tissue are manually analyzed. The novel automatic fluorescencesignal counting system, however, enables the pathologist to analyze avery large number of nuclei and obtain a thorough understanding of thebiopsy tissue. This is advantageous considering that only one biopsytissue sample is typically taken from the breast 11. Conventional manualanalysis methods do not achieve the best diagnostic and prognosticresults because only a small number of nuclei from a single biopsytissue sample of the patient's breast are analyzed. The analysis system,however, is able automatically to count an arbitrarily large number ofcell components. The biopsy tissue sample is sliced and made into manyslides, for example, up to one thousand slides. FIG. 1 shows the nthslide 12 of the one thousand slides.

FIG. 2 shows how the fluorescence signal counting system scans thebiopsy tissue 10 on slide 12 at many locations. For example, the systemcan scan biopsy tissue on a typical slide at up to two hundredlocations. At each location of the scans, the system generates multipleimage slices at different depths of the z-axis. The multiple imageslices are obtained using a confocal microscope. The stacks of imageslices are called “z-stacks”. FIG. 2 shows the top image slice 13 of az-stack 14.

FIG. 3 shows the many nuclei apparent in each image slice of z-stack 14.Hundreds of nuclei are typically apparent in each image slice. Forexample, a non-cancerous nucleus 15 is one of the nuclei apparent in topimage slice 13. In this embodiment of the system that countsfluorescence signals present in breast tissue marked using the FISHtechnique, identifying the cell components within each nucleus is ofprimary importance. Therefore, only the membranes of the nuclei areshown in FIG. 3, as opposed to the cell membranes.

FIG. 4 shows certain cell components of a non-cancerous nucleus 15,which has been stained with DAPI (4,6-diamidino-2-phenylindole). Thecell components include centromeres and genes present on humanchromosome number seventeen 16. In the FISH technique, fluorescencesignals are emitted by the marked cell components. In this embodimentthat is related to the diagnosis of breast cancer, genes present onchromosome number seventeen are analyzed. The nucleus of a normal celltypically contains two copies of chromosome number seventeen. Genes thatare to be analyzed are marked in the FISH technique with fluorescentprobes. The marked genes then appear as brightly colored areas whenviewed with a fluorescent microscope. Various parameters of the markedgenes are then determined, such as number, size, shape, distance betweenthe marked genes and distance between a marked gene and the nuclearmembrane. In this embodiment related to the diagnosis of breast cancer,the FISH technique is used to analyze the Her-2/neural (Human epidermalgrowth factor receptor-2) gene in breast biopsy tissue in order toprovide a diagnosis and prognosis for breast cancer. In cancerous cellsof the human breast, a high degree of amplification of the Her-2/neugene produces an overexpression of the corresponding protein. Thus,detecting an amplification of the Her-2/neu gene and a correspondingoverexpression of the Her-2/neu protein is an indication of a poorprognosis for mammary carcinoma. The diagnosis of metastatic breastcancer from Her-2/neu overexpression is all the more useful because ofthe development of drugs that directly target the Her-2/neu protein andare specially suited to combat this type of cancer, such as Trastuzumab(Herceptin). These anti-cancer drugs are quite expensive. By betterdiagnosing this particular type of cancer, medical costs can be saved byadministering these expensive drugs only to those cancer patients whohave this particular type of cancer. In addition, by detecting theamplification of genes other than Her-2/neu, the analysis systemsupports the administration of personalized medication when new drugsare developed. Finally, personalized dosages based on a particularpatient's biopsy tissue can also be administered.

The degree of amplification is determined by the ratio of the number offluorescence signals from the Her-2/neu gene to the number offluorescence signals that mark the centromere of each chromosome numberseventeen on which the Her-2/neu genes are located. FIG. 4 shows twocopies of chromosome number seventeen 16. In FIG. 4, each chromosomenumber seventeen 16 has two Her-2/neu genes and one centromere 17. Byconvention, the Her-2/neu genes that are marked with the fluorescentprobe LSI-HER-2/neu are made to appear pink or orange colored whenviewed with a fluorescent microscope. The microscopic images areacquired in the gray scale and are then colorized using acceptedconversions into the RGB scale for ease of viewing. Each centromere 17of chromosome number seventeen that is marked with the fluorescent probeCEP-17 typically appears green. FIG. 4 shows two green fluorescencesignals 18, each indicative of one copy of chromosome number seventeen16. A pink fluorescence signal 19 illuminates chromosome numberseventeen 16 at each location of the Her-2/neu gene. In a non-cancerouscell, there is typically one Her-2/neu gene on each chromosome numberseventeen 16. The Her-2/neu gene is considered not to be amplified wheneach chromosome number seventeen 16 has only one Her-2/neu gene. TheHer-2/neu gene is highly amplified where there are more than fourHer-2/neu genes on each chromosome number seventeen 16.

In one accepted diagnostic procedure, one hundred nuclei of a biopsytissue sample are analyzed in order to calculate the degree ofamplification of the Her-2/neu gene in the biopsy tissue sample. In eachof the one hundred nuclei, the fluorescence signals from each Her-2/neugene and each chromosome number seventeen are counted. The degree ofamplification of the biopsy tissue sample is then categorized as being(i) not amplified, (ii) moderately amplified or (iii) highly amplifiedbased on the following criteria. The sample is not amplified if lessthan ten percent of the nuclei have more than four signals 19 indicativeof the Her-2/neu gene. The sample is highly amplified if more than tenpercent of the nuclei have more than ten signals 19 indicative of theHer-2/neu gene. And the sample is moderately amplified if more than tenpercent of the nuclei have both (i) more than four but less than orequal to ten signals 19, and (ii) a quotient of signals 19 to signals 18(indicative of chromosome number seventeen) of greater than two. Foradditional information on diagnostic procedures based on countingsignals indicative of the Her-2/neu gene, see Pauletti et al.,“Detection and quantitation of HER-2/neu gene amplification in humanbreast cancer archival material using fluorescence in situhybridization,” Oncogene, 13:63-72, Jul. 4, 1996, which is incorporatedherein by reference. For different types of cancer, and even fordifferent types of breast cancer, the ranges for the degrees ofamplification differ.

In another accepted diagnostic procedure for the type of breast cancerthat responds to Trastuzumab (Herceptin), the fluorescence signals fromeach Her-2/neu gene and each chromosome number seventeen in one hundrednuclei are also counted. The ratio of Her-2/neu fluorescence signals 19to chromosome number seventeen fluorescence signals 18 for each nucleusis then calculated. Finally, the average of the ratios is calculated.The pathologist uses the average of the ratios to develop a diagnosis ofbreast cancer.

There are, however, complicating factors that have hindered conventionalcounting methods from obtaining an accurate count of the number ofHer-2/neu fluorescence signals 19 per nucleus. The novel automaticfluorescence signal counting system overcomes these complicatingfactors. For example, the analysis system operates as a computer programand is not prone to fatigue and loss of concentration. Moreover, thesystem recognizes when the components of a nucleus have already beencounted and does not recount that nucleus, which would result in adouble count. In addition, by recognizing which nuclei have already beencounted, not-yet-counted nuclei are not missed. Most important, thesystem can determine whether a fluorescence signal represents a singlegene or multiple overlapping genes. The system can also distinguishbright spots on the image slices that are artifacts and not fluorescencesignals at all.

FIG. 5 illustrates how the three-dimensional nature of the cellcomponents complicates the counting and can lead to incorrect results inconventional methods that count based on two-dimensional images andtwo-dimensional composites of three-dimensional information. The cells,nuclei and cell components of the biopsy tissue are present at differentdepths of the z-dimension in each slide. A fluorescence signal from afirst gene that lies directly above a second gene in the z-dimensionwill overlap any fluorescence signal from the second gene. The signalfrom the second gene will go unnoticed in a two-dimensional image in thex-y plane. In addition, the signal from a centromere will cover up thesignal from a gene that lies directly below the centromere.

FIG. 5 shows a nucleus 20 in a first cell 21, as well as a nucleus 22 ina second cell 23. Most of nucleus 20 of first cell 21 is at a higherdepth in the z-stack than is nucleus 22 of second cell 23. The analysissystem has generated three image slices of z-stack 14 that cut throughthe nuclei of first cell 21 and second cell 23. Nucleus 20 includes afirst gene 24 that overlaps a lower-lying second gene 25.

FIG. 6 illustrates how the cell components of first cell 21 and secondcell 23 of FIG. 5 would look when viewed from above in only twodimensions. FIG. 6 shows a composite 26 of the image slices of FIG. 5.In this view, the Her-2/neu fluorescence signals 19 emanating fromnucleus 20 partially overlap each other. Thus, from the x-y perspective,there are overlapping Her-2/neu genes 27. FIG. 6 also shows Her-2/neugenes 28 that are partially covered by a centromere. The analysis systemis able accurately to count the Her-2/neu genes because the analysissystem generates a data network based on the pixel values of multipleslices in the z-dimension.

FIGS. 7A-C show actual microscopic images of nuclei, as well as genesand centromeres within the nuclei. FIG. 7A is a digital image acquiredusing a blue light channel to emphasize the stained nuclear membranes.Even using a blue light channel, it is difficult using conventionalcounting methods to distinguish the multiple overlapping nuclei from oneanother in FIG. 7A. FIG. 7B is a digital image acquired using a redlight channel to emphasize the pink Her-2/neu fluorescence signals 19.FIG. 7C is a digital image acquired using a green light channel toemphasize the green chromosome number seventeen fluorescence signals 18.FIGS. 7B and 7C illustrate the difficulty in using conventional countingmethods to determine which Her-2/neu genes and which chromosomes numberseventeen fall within each particular nucleus of FIG. 7A. From thetwo-dimensional z-y perspective of FIGS. 7B, for example, it isdifficult to determine the depth of each pink Her-2/neu fluorescencesignal 19 and therefore to identify the nucleus to which each Her-2/neugene belongs.

FIG. 8 shows actual microscopic images of nuclei containing greenfluorescence signals 18 that mark the centromeres of chromosome numberseventeen and pink Her-2/neu fluorescence signals 19. The left mostnucleus is from a non-cancerous cell. The nucleus has two greenfluorescence signals 18 and two pink Her-2/neu fluorescence signals 19.The two green signals 18 indicate the presence of two copies ofchromosome number seventeen. The two pink signals 19 indicate thepresence of two copies of Her-2/neu genes in the same nucleus. Thus, theratio of Her-2/neu genes to chromosome number seventeen is 1.0, and theHer-2/neu gene is not amplified. The two nuclei on the right of FIG. 8are from cancerous cells; each has clumps of overlapping pink Her-2/neufluorescence signals 19 that are difficult to distinguish from oneanother in the x-y perspective. The second nucleus from the right hasthree green signals 18 and approximately 13 pink signals 19. The ratioof Her-2/neu genes to chromosome number seventeen is thereforeapproximately four, indicating that the Her-2/neu gene is highlyamplified.

FIG. 9 is a diagram that represents part of a data network based onpixel values 29 of the three image slices of FIG. 5 that cut throughfirst cell 21 and second cell 23. The pixel values of top image slice 13that depict first gene 24 are linked to a first object 30. The pixelvalues of a lower image slice 31 that depict second gene 25 are linkedto a second object 32. Both first object 30 and second object 32 arelinked to a superordinated object 33. By linking objects obtained fromimage slices at different depths of the z-dimension, additionalinformation can be extracted from the biopsy tissue 10 than would beavailable by analyzing images of the tissue only from the x-yperspective. For example, a distance 34 between first gene 24 and secondgene 25 can be determined. The distance 34 between first gene 24 andsecond gene 25 is also illustrated in FIG. 5. In addition, the volume ofa cell component that intersects multiple image slices can also bedetermined.

Objects associated with pixel values on the same image slice can also belinked. Objects can be linked not only based on their relative positionson the image slice. For example, objects that represent similar nucleican be linked, such as nuclei with similar areas, shapes orbrightnesses. Nuclei can be linked based on their concavity orconvexity, for example. This would permit all Her-2/neu genes located innuclei having a similar shape to be linked. Then the analysis systemperforms a specific algorithm on all of the linked genes. In anotherexample, the distance between nuclei in one image slice is determined bylinking nuclei objects in that level. Where the distance is determinedto be small because the nuclei are clustered, the analysis system canapply a more extensive algorithm in the z-dimension to separate anddistinguish the nuclei from one another. In yet another example, thearea or volume of a gene signal relative to the area or volume of theassociated nucleus can be determined by linking objects from the sameimage slice as well as from different levels.

The novel automatic fluorescence signal counting system generates notonly a data network based on pixel values of image slices, but also aclass network and a process hierarchy. The class network defines thecharacteristics of objects that that will be classified as various cellcomponents. The process hierarchy includes process steps that controlthe flow of the analysis and calculations performed on the pixel valuesand objects.

FIG. 10 is a simplified diagram of a computer-implemented networkstructure 35 used by the analysis system to count and analyzefluorescence signals obtained using the FISH technique. Networkstructure 35 is generated by the analysis system and an associatedcomputer program. The associated computer program is called theCognition Program. The network structure 35 includes a data network 36,a class network 37 and a process hierarchy 38. In the example of FIG.10, data network 36 includes a first data table 39 and a second datatable 40. The table data values in data tables 39-40 are in the form ofboth numbers and text.

In this embodiment, some of the table data values are digital pixelvalues from image slices of biopsy tissue 10, while other table datavalues describe the patient from whom the biopsy was taken. Thus, someof the table data values are floating-point values representing thespectral intensity of individual pixels of the image slices. The othertable data values are items of metadata relating to whether the patientmight have breast cancer. Examples of such information include thepatient's gender, age, weight, height, blood values, prescribedmedications, number of children, family history of ailments, whether thepatient breast-fed her children, whether the patient smoked or useddrugs. In FIG. 10, a first value 41 and a second value 42 are spectralintensity values from the image slices, whereas a third value 43 is anitem of metadata, such as the weight of the patient. In this example,the pixel values of first data table 39 correspond to the top imageslice 13, and first value 41 corresponds to a pixel value of image slice13 that depicts first gene 24. Similarly, the pixel values of seconddata table 40 correspond to the next lower image slice 31, and secondvalue 42 corresponds to a pixel value of image slice 31 that depictssecond gene 25.

In this embodiment, network structure 35 is used by the analysis systemto count nuclei and other cell components. The visual inspection ofslides and the manual counting of cell components is time consuming andlabor intensive. Because of the low prevalence of fluorescence signalsfrom highly amplified Her-2/neu genes in most of the slides viewed bythe pathologist, tedium can cause the pathologist to overlook highlyamplified Her-2/neu genes when they are present. Network structure 35and the analysis system that generates network structure 35 help thepathologist to avoid overlooking and miscounting Her-2/neu genes.

FIG. 11 is an example of image slices that are analyzed by networkstructure 35 and the Cognition Program. For example, the three imageslices of FIG. 11 are analogous to the three image slices of FIG. 9. Thetop image slice of FIG. 11 corresponds to first data table 39, and themiddle image slice of FIG. 11 corresponds to second data table 40. Inother embodiments, second data table 40 also includes metadata relatingto the patient. Network structure 35 is used to identify and count cellcomponents, such as nuclear membranes, centromeres and genes. Toidentify these three cell components, the analysis system filters outseparate wavelengths of light that make up each image slice. The imagedata is typically acquired in a gray scale and then interpreted into anRGB scale by convention. Color schemes other than red-green-blue canalso be used, for example, the hue-saturation-brightness (HSB) colorscheme. For example, Her-2/neu fluorescence signals 19 could beinterpreted as yellow instead of pink. In this example, nuclearmembranes are most apparent in the gray scale corresponding to bluelight; the marked centromeres of chromosome number seventeen are mostapparent through a green filter; and the pink Her-2/neu fluorescencesignals 19 pass through a red filter. Thus, for each image slice, theanalysis system generates a pink image slice showing the Her-2/neugenes, a green image slice showing the centromeres of each chromosomenumber seventeen, and a blue image slice showing the membranes of thenuclei. The analysis system links the pixel values of the pink imageslice to objects classified in the class hierarchy as genes. Pixelvalues of the green image slice are linked to objects classified ascentromeres, and pixel values of the blue image slice are linked toobjects classified as nuclear membranes.

FIG. 12 shows the red, green and blue image slice components that makeup the image slices of FIG. 11. On the left of FIG. 12, three red imageslices show pink Her-2/neu fluorescence signals 19. In the middle ofFIG. 12, green image slices show green fluorescence signals 18indicative of chromosome number seventeen. On the right of FIG. 12, blueimage slices show the membranes of the nuclei that contain thechromosomes with the Her-2/neu genes.

Returning to FIG. 10, data network 36 also includes objects and links.In this example, first value 41 is linked by a first link 44 to firstobject 30. First object 30 in FIG. 10 corresponds to first object 30 inFIG. 9. Second value 42 is linked to second object 32. First object 30is linked by a second link 45 to object 33. Second object 32 is alsolinked to object 33. Class network 37 includes a class 46, a subclass 47and a second subclass 48. Class 46 is linked to subclass 47 and tosecond subclass 48. In addition, class 46 of class network 37 is linkedto object 33 of data network 36. In this example, class 46 is the classfor Her-2/neu genes. And subclass 47 is linked to second object 32. Inthis example, subclass 47 is the class for Her-2/neu genes that areoverlapped by other genes. Process hierarchy 38 includes a process step49. Process step 49 in turn includes a domain specification 50 and analgorithm 51. Algorithm 51 is linked by a third link 52 to the thirdvalue 43 of first data table 39. Domain specification 50 is linked by afourth link 53 to the object 33. Thus, an algorithm of a process step inprocess hierarchy 38 is linked to metadata in data network 36, and adomain specification of a process step in process hierarchy 38 is linkedto an object in data network 36.

FIG. 13 is a flowchart illustrating steps 54-59 of a method by which theanalysis system uses network structure 35 and the Cognition Program toperform computer-aided detection (CAD) of cell components that theanalysis system then counts. In other embodiments described below,network structure 35 is used to detect objects other than cellcomponents. The steps of FIG. 13 will now be described in relation tothe operation of network structure 35 of FIG. 10.

In first step 54, a user of the analysis system specifies class network37 by defining the likelihood that objects of data network 36 willbelong to each particular class of class network 37. The user of theanalysis system is, for example, a research doctor who is applying hisexpert knowledge to train the analysis system in a specification mode.Such a research doctor could be a research pathologist working at apharmaceutical company, for example. In addition to the research doctor,pathologists then also use the analysis system in an execution mode.

FIG. 14 shows class network 37 of FIG. 10 in more detail. Class network37 includes classes linked to subclasses that describe what the userexpects to find in the image slices included in first data table 39 andsecond data table 40. Thus, in this example, the classes and subclassesof FIG. 14 describe what the user expects to see in the image slices ofFIG. 11. The user starts by giving each class a name. In this example,the user has specified a background class 60, an image border class 61and a separate class for each of the N cells having a nucleus that is tobe counted. For example, cell number one has a subclass 62 for theHer-2/neu genes of the cell that are marked with pink Her-2/neufluorescence signals 19. Cell number one also has a subclass 63 forgreen fluorescence signals 18 that mark the centromeres of chromosomenumber seventeen of the cell. The user has also given subclass 63 itsown subclass 64 for those centromeres that overlap Her-2/neu genes. Theuser specifies a helper class to categorize cell components whoseidentity is unknown at the beginning of the analysis.

The user also specifies categories of metadata. In this example, classnetwork 37 includes a class for patient data and subclasses specifyingthe types of patient data. The user has specified subclasses for thepatient's age, weight, height, number of children, whether the patientbreast-fed her children, the patient's family history of ailments, thepatient's blood values 65, and whether the patient smoked.

Each class may have an associated membership function that defines theprobability that an object of data network 36 will belong to theparticular class. The membership functions do not define whether anindividual pixel value belongs to a class. Rather, each object is agroup of pixels linked to the object, and the user specifies themembership function by defining the properties that the object must haveto belong to the class. Examples of such properties include the area,shape, color and texture of the object. The area of an object may bedetermined, for example, by the number of pixels linked to the object.An item of metadata may also be a variable in a membership function. Forexample, the texture of an object that belongs to the class “nuclearmembrane” may be different if the age and weight of the patient are overcertain thresholds.

FIG. 15 is a flowchart that illustrates the substeps of step 54 of FIG.13 in more detail. In a substep 66, a link type for a link between twoclasses or subclasses is specified. Links that are not specified insubstep 66 can also be specified later in step 56 of FIG. 13.

In step 55 of FIG. 13, the user specifies process hierarchy 38 of FIG.10. The user specifies not only the individual process steps, but alsothe order in which the process steps are to be executed in the executionmode of the analysis system. Thus, each process hierarchy has a rootprocess step linked to other process steps. The process steps in turnmay be linked to substeps.

FIG. 16 shows process hierarchy 38 of FIG. 10 in more detail. Processhierarchy 38 includes a root process step 67 named “FISH 3D Analysis”with a domain specification 68 and an algorithm 69. In this example, theuser has specified four process steps 70-73 linked in a specific orderto root process step 67. The first process step 70 “FISH Mamma 2D” has asub-process step 74 named “Image Border”. The third process step 72“Link Objects, Slices, Nuclei and Signals” has four sub-process steps75-78, three of which have their own substeps. Substep 75 “3D Processes”has a first substep 79 named “Prepare Slices” and a second substep 80“Classify Slices”. Substep 76 “Find Overlap” has a substep 81 named“Link Nuclei Using Overlap Calculation”. Substep 77 “Some Corrections”has a substep 82 named “Chromosome 17 Signal Overlapping Her-2 Signals”.

For each process step or sub-process step, the user has the option ofspecifying a domain and an algorithm. FIG. 16 shows that the user hasspecified a domain 83 for the process step 72 and a domain 84 for thesub-process step 82. The domain specifies classes that define theobjects of data network 36 upon which the algorithm is to operate at runtime in the execution mode. FIG. 16 also shows that the user hasspecified an algorithm 85 for the process step 72 and an algorithm 86for the sub-process step 81.

FIG. 17 is a screenshot of a view of a graphical user interfacegenerated by the analysis system. The screenshot includes a middlewindow containing an outline representation of the process hierarchy 38of FIG. 16. Corresponding process steps from FIG. 16 are labeled in FIG.17. The substeps of subprocess step 74 have been expanded, and thesubsteps of the other process steps are hidden in this graphical userinterface. The user can add and delete process steps from the processhierarchy 38 by using pull-down windows that appear in response to aright mouse click.

FIG. 18 is a flowchart that illustrates the substeps of step 55 of FIG.13 in more detail. In a substep 87, the user specifies the domain of theprocess step that the user has created. In a substep 88, the userspecifies an object filter for the domain. For example, from the objectswithin the domain, the object filter passes to the algorithm all ofthose objects that are linked to fewer than sixteen pixel values. In adecision substep 89, the analysis system queries the user as to whetherthe domain is navigational. The user is queried by a dialogue in apop-up window. The domain is navigational when the objects to beoperated upon are defined based on how they are linked to other objects.For example, a domain may include only those subobjects that are linkedby a certain type of link to a parent objects. If the domain isnavigational, in a substep 90 the user specifies the link types thatdefine the vicinity of the parent object. If the domain is notnavigational, in a substep 91 the user specifies an object container forthe objects that are to be operated upon by the algorithm of the processstep. For example, the object container can be all of the objects at aspecified object level. The level of objects linked directly to tabledata values is referred to as object level zero. Objects linked directlyto objects in object level zero are considered to be in object levelone, and so forth.

In a substep 92 of step 55 of FIG. 13, the user specifies the algorithmthat will operate on the objects specified in the domain. The user canchoose preformulated algorithms from a database of algorithms accessedby the analysis system. For example, some algorithms are used for thesegmentation of objects. Other algorithms are used for computation, suchas for statistical calculations or to calculate the area of pixelslinked to an object.

FIG. 19 shows a screenshot of a pop-up window generated by the analysissystem to assist the user to specify the algorithm as described insubstep 92.

Returning to FIG. 18, in a substep 93, the user specifies a breakcondition at which the algorithm stops operating on objects. Forexample, the algorithm may be iterative and operate on a group ofobjects a predetermined number of times, as defined by the breakcondition.

In step 56 of FIG. 13, the user then specifies various types of links.In network structure 35, links can be between objects, between classes,and between process steps (collectively referred to here as nodes). Inaddition, there can be links between a class and an object, between aclass and a process step, between a process step and an object, betweena process step and table data, and between an object and table data. Thelinks between a class and an object, between a process step and anobject, and between an object and table data exit in network structure35 only at run time during the execution mode of the analysis system.The user then uses the link types to define the relationship between thenodes of the class network and process hierarchy that the user specifiesin the specification mode. In addition, the user uses the link types todefine the relationship between the objects of the data network and theother nodes of network structure 35 that are to be generated at runtime.

FIG. 20 shows representations of various types of links 94-103 innetwork structure 35. The links describe the relation between theobjects, classes and process steps. The most elementary types of linksare either (i) exchange-relation links or (ii) relation links.Exchange-relation links describe an abstract, material or communicativeexchange between nodes. Relation links, on the other hand, describe therelationship between nodes depending on relational contents. Whereinformation is structured hierarchically, links are further subdividedinto two groups. The first group links nodes at different hierarchylevels. The second group links nodes at the same hierarchy level.

Link 94 represents an exchange-relation link that connects nodes atdifferent hierarchy levels. Link 94 represents the relationship betweena larger, super-ordinated node A and a smaller, subordinated node B.Thus, link 94 represents a change in scale of information and denotes “Bis part of A”. Links 95-97 are exchange-relation links that connectnodes in the same hierarchy levels. These links do not represent achange in scale of information and denote “B is an output quantity ofA”. For example, the link 97 denotes “B is an attribute of A”.

Link 98 represents a relation link that connects nodes at differenthierarchy levels and thus performs a scale change. Link 98 denotes “B ingeneral is A”. Links 99-102 represent relation links that connect nodesin same hierarchy level. Link 100 denotes “A is locally adjacent to B”;link 101 denotes “A is similar to B”; and link 102 denotes “A isfollowed by B”.

Link 103 represents a link that connects nodes that are capable ofcarrying out certain operations on other nodes and links. For example, anode connected to link 103 can generate new nodes or links and can alsodelete a node or a link. Link 103 denotes “B is function of A”. Foradditional information on types of links in a semantic networkstructure, see U.S. patent application Ser. No. 11/414,000 entitled“Situation Dependent Operation of a Semantic Network Machine,” filed onApr. 28, 2006, which is incorporated herein by reference.

Although in the embodiment of FIG. 13 the link types are specified instep 56 after the class network and the process hierarchy are specified,in other embodiments the link types are specified before the classnetwork and the process hierarchy are specified.

FIG. 21 illustrates the condition of computer-implemented networkstructure 35 in the specification mode after the user has specifiedclass network 37 in step 54, process hierarchy 38 in step 55, and thelink types in step 56. In this example, in the specification mode theuser has specified a link 104 between algorithm 86 and subclass 65(Blood Values) of class network 37. The user specifies link 104 byspecifying that algorithm 86 determines which nuclei overlap one anotherdepending on the blood values of the patient. In the specification mode,the user has also specified links 105 between domain specification 84and subclasses of class network 37, including a subclass 64 (CentromereOverlapping a Gene). The user specifies links 105 by specifying thatdomain 84 includes the objects that are determined in the execution modeto belong to subclass 64 and the analogous subclasses of the othercells.

In step 57 of FIG. 13, the analysis system acquires the values of firstdata table 39 and second data table 40. In this example, the pixelvalues of the images of FIG. 11 are generated by a confocal microscope.In other embodiments, the images are generated by an X-ray mammographydevice, a computed tomography (CT), an ultrasound imaging device, or amagnetic resonance imaging (MRI) device. The confocal microscopeincludes an image digitizer that converts the acquired images intodigital images. In other embodiments, physical film of microscopicimages is sent through a film digitizer to obtain the pixel values ofthe data tables. In yet other embodiments, the microscopic images areproduced directly in digital format. The digital pixel values of thedata tables indicate both the brightness levels and the color in thespace domain of the images of FIG. 11. Thus, each image slice iscaptured using multiple wavelength channels, in this examplecorresponding to red, green and blue. Metadata values are also acquiredin step 57. In this example, some of the metadata is in text format,such as the identity of medication prescribed for the patient. Othermetadata, such as the patient's weight, is in the form of a digitalnumber.

In step 58 of FIG. 13, the analysis system runs in the execution modeand generates data network 36 by selectively linking table data valuesto objects according to the class network and the process hierarchy.While the classes of class network 37 describe what the user expects tofind in the table data values, the objects reflect what the analysissystem actually finds in the table data values. At run time in theexecution mode, the analysis system executes the process steps asspecified in process hierarchy 38. Each object is generated by linkingto that object pixel values having similar characteristics, such asbrightness or the difference in brightness between a pixel and itsneighbors. Thresholds of brightness of pixels that are associatedtogether can be obtained from a gradient histogram of the pixel valuesin the digital image. The objects are then linked together into classesaccording to the membership functions of the classes. Thus, classes arelinked to objects at run time. In addition, classes and process stepsare linked to table data at run time.

FIG. 22 illustrates the condition of computer-implemented networkstructure 35 in the execution mode after the analysis system hasgenerated data network 36. Various classes of class network 37 have beenlinked to objects in data network 36 that belong to the classes. Forexample, the subclass 62 specifying a Her-2/neu gene of cell number oneis linked by a link 106 to an object 107, which in turn links pixelvalues of the image slice of second data table 40. Similarly, subclass62 of cell number one is linked by a link 108 to an object 109 thatlinks pixel values of the image slice of first data table 39.

In addition, FIG. 22 shows that while the Cognition Program is running,links are also generated between classes, process steps, objects andtable data. For example, object 107 representing pixel values of anHer-2/neu gene in the image slice of second data table 40 is linked by alink 110 to object 109 representing pixel values of the same Her-2/neugene in the image slice of first data table 39. Thus, the analysissystem recognizes that the pixels values belonging to both object 107and object 109 are associated with the same Her-2/neu gene.

Moreover, algorithms are linked to table data values. For example,algorithm 86 is linked by link 104 to class 65 (Blood Values) in thespecification mode. In the execution mode, class 65 is linked by a link111 to an object 112 for patient metadata. Thereafter in the executionmode, algorithm 86 is linked to an item of metadata 113 that contains avalue representing the patient's blood values. Network structure 35 isshown in FIG. 22 as a cognition network 114 when links are presentbetween classes, process steps, objects and table data at run time inthe execution mode.

FIG. 23 illustrates objects that have been linked to table data valueshaving similar characteristics. In addition, objects are linked to otherobjects according to the membership functions that define the classes.The table data values of FIG. 23 are arranged to illustrate that theyare pixel values of a digital image. For example, one factor of amembership function is the area occupied by the pixels that make up theobject. In one example, the area is calculated as being proportional tothe number of pixel values linked to the object.

FIG. 24 illustrates objects linked to other objects in data network 36.The objects are linked by the link types shown in FIG. 20. In theexecution mode, an object that belongs to a class is linked to anotherobject that belongs to another class when the two classes are linkedtogether in class network 37.

FIG. 25 is a flowchart that illustrates the substeps of step 58 of FIG.13 in more detail. In a substep 115, the class network that is specifiedin step 54 of FIG. 13 is loaded into a script execution engine of theCognition Program. In a substep 116, the process hierarchy that isspecified in step 55 of FIG. 13 is loaded into the execution engine. Ina substep 117, a data set N of the table data values acquired in step 57of FIG. 13 is loaded into the execution engine.

In a substep 118 of step 58 of FIG. 13, the process steps specified instep 55 of FIG. 13 are executed on the data set N. In substep 119, theuser has the option to run the Cognition Program in an interactive mode.In the interactive mode, the results of the computer-aided detection aredisplayed to the user, such as a research doctor. If the user is notsatisfied with the results, the user can edit the classes of classnetwork 37 or the process steps of process hierarchy 38 and immediatelyre-execute the process steps on the data set N. The user can edit theprocess steps using the graphical user interface and the script editorof the Cognition Program.

FIG. 26 is a screenshot of one view of the graphical user interfacegenerated by a view module of the Cognition Program. The screenshotincludes a window containing three image slices (top, middle and bottom)at three successive depths of in the z-dimension. The image slices showcell nuclei. The user can edit the class network 37 and the processhierarchy 38 using the windows on the right of the screenshot so thatthe target region recognized by the particular process step being editedis satisfactory. For example, by right mouse clicking on a process stepin the lower right window, a pup-up window appears with a dialogueasking the user whether he wishes to add a sub-process step or append aprocess step below the clicked process step. The user is then asked tochoose a domain and an algorithm for the new process step. Existingprocess steps can also be edited.

The user can also add or edit classes using the upper right window. Aclass is also added by right mouse clicking and responding to thequeries in the pop-up window. The user is asked to name the new classand enter properties of objects that belong to the class, such as color,area, asymmetry, density and the angles along the border of the object.Thus, the Cognition Program can also analyze color digital images. Inthis embodiment of the analysis system that automatically countsfluorescence signals present in biopsy tissue marked using the FISHtechnique, the Cognition Program analyzes images slices acquired usingred, green and blue color channels as shown in FIG. 12. In the imageslice shown in the graphical user interface in FIG. 26, however, thecolors of the nuclei do not correspond to the spectral colors used toacquire each image slice. Instead, the user has assigned each nucleus toa different class. Then, for ease of viewing, the user has assigned adifferent color to each class that is shown on the graphical userinterface.

As part of creating a class, the user also defines a membership functionfor objects that belong to the class. For example, the user can definean “asymmetry function” as part of the membership function. Theasymmetry function describes the shape of the pixels that make up anobject by approximating an ellipse. For example, the user can use theasymmetry function to classify nuclei objects. The numerator of theasymmetry function describes the long axis of the ellipse, and thedenominator describes the short axis. A shape of pixels thatapproximates a circle has an asymmetry value of one. An elongated shapeof pixels has an asymmetry value much greater than one. The user canalso define a “density function” to classify nuclei objects. The densityfunction is the square root of the area of the pixels divided by thelength of the border around the pixels that comprise the object. Theasymmetry function and the density function can be used in the diagnosisof breast cancer. The shape of a nucleus in a cancerous cell isdifferent from the shape of a nucleus in a normal cell.

FIG. 27 is a screenshot of another view of the graphical user interfacegenerated by the view module of the Cognition Program. The screenshotincludes a window on the left showing a portion of an image slice inwhich an Her-2/neu gene in a nucleus has been outlined. A window on theright provides information about each Her-2/neu gene in the image slice,including the outlined Her-2/neu gene. For example, the analysis systemindicates that the outlined Her-2/neu gene has an area of nineteenpixels. The analysis system also indicates the x-y coordinates of theoutlined Her-2/neu gene within the image slice.

FIG. 28 is a screenshot of yet another view of the graphical userinterface showing what appears to be an outlined Her-2/neu gene in thewindow on the left. A window on the right provides information about theoutlined object. In this example, the analysis system indicates that theoutlined object has an area of 582 pixels. Through the membershipfunction of the class for Her-2/neu genes, the analysis systemdetermines that the object is too large to be produced by an Her-2/neufluorescence signals 19. Therefore, the analysis system determines thatthe object is not an Her-2/neu gene, but rather an imaging artifact. Asin FIG. 27, the analysis system indicates the x-y coordinates of theoutlined artifact. From the x-y coordinates, the analysis systemdetermines that the Her-2/neu gene of FIG. 27 lies under and is coveredby the artifact of FIG. 28. Nevertheless, the analysis system is able toidentify the Her-2/neu gene of FIG. 27.

Because class network 37 and process hierarchy 38 are specified using aCognition Language (CL) based on the XML script language, class network37 and process hierarchy 38 can be edited without recompiling theCognition Program. Thus, the user can input a new membership function ofa new class at run time that defines whether the objects of data network36 will belong to the new class, and the process steps can be performedimmediately on the newly generated data network 36 without recompilingthe program instructions of the Cognition Program. The XML-basedCognition Language and the graphical user interface allow the user tomore quickly “train” cognition network 114 to recognize Her-2/genesmarked by fluorescence signals 19 and centromeres of chromosome numberseventeen marked by fluorescence signals 18. The ability to edit theclass network 37 and process hierarchy 38 at run time differentiates theCognition Program from conventional CAD schemes that cannot change theprocess of applying rules once the CAD scheme begins analyzing aparticular digital image. After the user of the Cognition Programdetermines that the results of the pattern recognition performed on adata set N are satisfactory, the process steps are automaticallyexecuted on the next data set N+1. The Cognition Program can thenautomatically perform the process steps on a large number of data sets,for example by performing the processing overnight. Reports aregenerated for each level of data, such as for all nuclei, for each imageslice, for each z-stack and for each slide.

The Cognition Program would typically not be run in the interactive modewhen the user is not a research doctor but rather a pathologist who isanalyzing a new patient's biopsy tissue. A pathologist would use theCognition Program with a class network and a process hierarchy that havealready been trained by the research doctor. In that case, all of theprocess steps of process hierarchy 38 would be executed on all of thedata sets, and the results would be saved for displaying as the finalresults in step 59 of FIG. 13.

FIG. 29 is a flowchart that illustrates in yet more detail additionalsubsteps of substep 118 of FIG. 25. FIG. 29 illustrates the process bywhich each process step of process hierarchy 38 operates on objectsspecified by the domain. In a substep 120, the domain that was specifiedin substeps 87-90 of FIG. 18 is generated. In a substep 121, theexecution engine retrieves the next object of the domain that is to beoperated upon. In a substep 122, the execution engine executes thealgorithm of the process step on the retrieved object. In a substep 123,the execution engine executes the algorithm of any sub-process steps onthe retrieved object. In a substep 124, the execution engine retrievesthe next process step of the process hierarchy so that substeps 120-123can be repeated for the domains and the algorithms of the next processstep and sub-process steps, if any.

FIG. 30 is a flowchart that illustrates in yet more detail additionalsubsteps of substep 120 of FIG. 29. FIG. 30 illustrates the process bywhich the domain that was specified in substeps 87-90 of FIG. 18 isgenerated at run time.

Returning to a final step 59 of FIG. 13, the Cognition Program outputsthe final results of the computer-aided detection based on the cognitionnetwork 114 that was generated using class network 37 and processhierarchy 38.

FIG. 31 shows cognition network 114 of FIG. 22 when data network 36 hasbeen generated from many data tables, each containing a digital image.Thus, FIG. 31 is a more detailed example of the diagram of FIG. 9 inwhich a data network is generated based on pixel values from multipleimage slices. By generating data network 36 from digital images obtainedfrom many parallel planar slices of biopsy tissue, for example, theCognition Program detects Her-2/neu genes in three dimensions within thebiopsy tissue.

FIG. 31 illustrates that in the specification mode a domainspecification is linked by a link 125 to a class. In the execution mode,the Cognition Program acquires table data values comprising the manydigital images that are slices of a three-dimensional data set. TheCognition Program then applies the membership function of the class tothe values of each of the digital images and determines that variousobjects generated from the many digital images belong to the class. Forexample, the Cognition Program determines that each of objects 126-131from digital images in data tables 132-136, respectively, belongs to theclass. The class is then linked to each of the objects 126-131. Forexample, a link 137 links the class to object 126, which in turn islinked to pixel values from a first digital image in data table 132. Atrun time, the domain specification is then also linked to the objectsthat belong on the class specified in the domain specification. Forexample, a link 138 links the domain specification to the object 126while the Cognition Program is running. This allows the algorithm of theprocess step to operate on all of the objects that comprise the3-dimensional object, in this example an Her-2/neu gene. Finally, eachof the objects 126-131 that belong to the class and that are generatedfrom the many digital images are linked to each other in data network36. For example, object 126, which is linked to pixel values from thefirst digital image, is linked by a link 139 to object 127, which islinked to pixel values from the second digital image. Because all of theobjects 126-131 are identified as belonging to the same class, thephysical characteristics of the class can be determined. For example,where the class is a gene clump or a nucleus, the analysis system candetermine the volume of the gene clump or nucleus.

FIG. 31 also illustrates that the Cognition Program has linked an objectgenerated from one image slice to two objects in an adjacent imageslice. Object 129 that is generated from the fourth image slice islinked to two objects of the same class generated from the fifth imageslice. Object 129 is linked by a link 140 to object 130 and by a link141 to a second object 131 from the fifth image slice. In this way, theCognition Program is able to detect 3-dimensional objects such as bloodvessels that fork into multiple portions in adjacent image slices.

Linking objects in multiple scans can also be used to track movementover time. Instead of the scans representing adjacent physical planes ofa physical object, multiples scans are analyzed that are acquired atdifferent times. In one example, the objects 126-131 belong to biopsytissue samples from the same patient that are taken at one-monthintervals. The change in shape in the nuclei and the change inamplification of the Her-2/neu genes is used to provide a prognosis ofthe patient's condition. In another example, the objects 126-131 belongto the class representing a cell. Digital images are taken of the cellat different time intervals in minutes. Movement can be tracked bylinking objects of the same class that are obtained from digital imagestaken at adjacent time intervals. Over the four time intervals at whichthe digital images of data tables 132-135 are taken, the cell describedby the class linked to objects 126-129 grows from four pixels to sevenpixels. Then after the fifth time interval, the cell divides into object130 with four pixels and object 131 with four pixels. The movement andchange in shape of cells and cell components need not be tracked inadjacent time intervals, but rather can be analyzed at irregular times.Movement of three- or N-dimensional objects can also be tracked. In oneexample, the fourth dimension analyzed is time, and the fifth dimensionis speed (change in time).

In another embodiment, the Cognition Program acquires the multipledigital images from a video movie instead of from multiple scans. Forexample, the video movie depicts movement of a bacterium, a cell or aninterphase nucleus. The Cognition Program can be used to detect onemoving cell from among multiple moving cells.

In yet another embodiment, the analysis system analyzes and correlatesimages acquired using different imaging and marking methods. Forexample, data tables 132-135 include images of the same biopsy tissuetaken using a microscope, an X-ray device, a computer tomograph, anultrasound imaging device, and a magnetic resonance imaging device.Thus, the analysis system provides a multi-modal display of imagesobtained from multiple spectral imaging techniques. The multi-modalimages are correlated to one another using the graphical user interfaceof the analysis system. In addition, the various images can be takenusing different biomarkers, such as FISH, chromogenic in situhybridization (CISH) and polyclonal antibody (NCL-Ki67p) labeling of theKi67 antigen. Thus, each image of the multi-modal display shows multiplebiomarkers. Images can also be taken using “multiplex” biomarkers thatmark different cell components in different ways. In addition, images ofmarked proteins, such as estrogen and progesterone, can also bedisplayed and analyzed. Objects that correspond to the same marked cellcomponents in the various multi-modal images are then linked in datanetwork 36. Diagnoses and prognoses can be improved by correlating theresults of analyzing the various images taken using different biomarkersand spectral analysis techniques.

FIG. 32 is a screenshot of a process hierarchy in another embodiment ofcognition network 114 that analyzes individual cells in a cell assay.Three-dimensional properties of the cells are analyzed using one hundredscans at different depths of an individual cell using a confocalmicroscope. During the “training” process when the Cognition Program isrun in the interactive mode as described by the substeps of FIG. 25, theprocess hierarchy of FIG. 32 is presented to the user in the lower rightwindow of FIG. 26.

FIG. 33 shows the output of step 59 of FIG. 13 as presented to the useron the graphical user interface for the embodiment of FIG. 32. Two3-dimensional cells from a cell assay are depicted in FIG. 33. In theimage of FIG. 33, target objects 142 that belong to the same class aredisplayed with the same color. In this example, target objects 142 aremarked mitochondria. The Cognition Program compares the mean volume ofthe marked mitochondria to the volume of surrounding cytoplasm.

In yet other embodiments, cognition network 114 analyzes cells that arein various forms of aggregation. Individual cells in wells in plates ofa cell-based assay can be analyzed. Grouped cells in a tissue formationassay can be analyzed. Tissue biopsy samples can be analyzed asdescribed above. And tissue micro arrays containing different types oftissue can be analyzed. A tissue micro array may, for example, includetissue from the skin, breast, lungs and heart that exhibits differenttypes of cancer. In this example, the analysis system is used tocorrelate the responses of the different cancer types in the differenttissues to particular dosages of drugs. Thus, the analysis system isused in pharmaceutical research to analyze tissue micro arrays. In themethod of FIG. 13, table data values are acquired in step 57 from imagesof cell tissue in multiple wells of a tissue micro array. The CognitionProgram is used to look for morphological changes in the cells. TheCognition Program is used to detect motion in the cell tissue byanalyzing multiple images taken of the same well after successive timeintervals. The Cognition Program determines how long it takes for thecells of the tissue to stop dividing when a specified dosage of a drugis placed in the well.

FIG. 34 is a listing of high-level lines of XML code that corresponds toa CL script that implements a class network and a process hierarchy forautomatically counting fluorescence signals present in biopsy tissuemarked using the FISH technique. The CL script was created and editedusing a graphical user interface similar to the one shown in FIG. 26.All of the lines of the XML code are present in an XML file entitled“FISH eCog converted XML.txt” and contained in the CD Appendix.

FIGS. 35A-F show more lines of the XML code of FIG. 34. The XMLdescription of selected classes of FIG. 14 and process steps of FIG. 16are identified by XML comments in FIGS. 25A-F. For example, FIG. 35Ashows an XML description 143 of the class “Cells” of FIG. 14. The class“Cells” is identified with the class ID of “7”. In addition, FIG. 35Ashows an XML description 144 of a Helper class of FIG. 14 labeled as“potential Her2/neu”. The Helper class is a temporary class andgenerates intermediate objects at run time. In addition to the Helperclass that is specified by the user, the analysis system also employsdynamically generated classes. These classes are generated at run timeand are deleted before the final result of classification is achieved.Intermediate objects are linked to the Helper class and to thedynamically generated classes. The intermediate objects are reclassifiedas the Cognition Program iteratively performs the process steps tooptimize the categorization of objects into the remainder of thespecified classes. Thus, performing the process steps of the processhierarchy is an adaptive process that is repeated and optimized on samedigital image.

FIG. 35D shows an XML description 145 of the domain specification of asub-process step of the sub-process step 70 (fish mamma 2D) of FIG. 16.The class ID “7” for the class “Cells” is listed under the list ofclasses <lClss> of the domain specification. The class ID in the domainspecification generates a link between the domain specification and theclass at run time. At run time, links are also generated from the domainspecification of the sub-process step to the actual objects of the datanetwork that are determined to belong to the class “Cells”.

The analysis system can also be applied on stains and biomarkers otherthan those that mark the Her-2/neu gene, such as IHC, CISH and Ki-67.Correlations between results from these other biomarkers and those fromFISH can be performed on a single specimen or tumor patient. Such anapplication is useful in applications of Translational Medicine and in a“Case Based Reasoning” method. Results from the analysis system can alsobe correlated to radiology image data in order to obtain a multiscale,multiresolution and multimodal computer-aided diagnosis. The disclosedmethod allows a user to correlate different image analysis results, suchas between morphological and molecular pathology and between pathologyand radiology. In addition, results at different clinical stages can becorrelated, such as between the time of analysis and the time oftreatment. In this way, detailed knowledge about the extracted objects,their mutual relationships and the entire tissue or tumor are generated.

The analysis system performs computer-aided detection (CAD) of cellcomponents in combination with a work station. The Cognition Programruns on the work station. For faster computation and analysis, anon-chip solution incorporates the process steps into hardwiredprocessing units of an application specific integrated circuit (ASIC) ora programmable logic device (PLD). The analysis system can also be usedin conjunction with “virtual microscopy” in which automated analysis ofslides is performed without manual involvement. Knowledge is extractedfrom a large amount of analyzed data that is acquired over many hours ordays. The analysis system is used to recognize patterns andrelationships in the large amount of data. For example, the analysissystem recognizes patterns in the distances between nuclei containingmultiple marked Her-2/neu genes and patterns in the shapes of markedHer-2/neu genes.

Compack Disc Appendix

The Compact Disc contains:

A) the file named FISH eCog converted XML.txt (684 KB, written to discon Nov. 30, 2006) which is an ASCII version of the XML code thatgenerates the visual representation of FIGS. 17 and 26-28; and

B) a file named CD Appendix Title Page.txt, which has 319 bytes and waswritten to disc on Nov. 30, 2006.

Although the present invention has been described in connection withcertain specific embodiments for instructional purposes, the presentinvention is not limited thereto. For example, although embodiments ofthe Cognition Program and computer-implemented network structure havebeen described above in relation to the computer-aided analysis andcounting of fluorescence signals present in biopsy tissue marked usingthe FISH technique, the Cognition Program and network structure canequally be applied to detecting and analyzing target patterns in othermicroscopic images. In addition to analyzing images acquired withconventional confocal microscopy, microscopic images acquired in theinfrared light band can also be analyzed. The Cognition Program andnetwork structure can also be used to detect and analyze anatomicalregions such as the human brain, lungs and breasts, as well asmicroorganisms, non-living cells and the cells of plant. Thus, theanalysis system can be used in environmental applications. Accordingly,various modifications, adaptations, and combinations of various featuresof the described embodiments can be practiced without departing from thescope of the invention as set forth in the claims.

1. A method comprising: specifying a class network; specifying a processhierarchy; acquiring pixel values of an image that includes cellcomponents, wherein certain of the cell components are marked usingfluorescence in situ hybridization; generating a data network based onthe class network and the process hierarchy; and counting the markedcell components using the data network.
 2. The method of claim 1,wherein the data network includes an object that is linked to some ofthe pixel values, and wherein the object corresponds to a marked cellcomponent.
 3. The method of claim 2, wherein the marked cell componentis an Her-2/neural gene.
 4. The method of claim 2, wherein the markedcell component is a centromere of a chromosome.
 5. The method of claim2, wherein the marked cell component is a nucleus of a cell.
 6. Themethod of claim 1, wherein one of the cell components has a volume,further comprising: determining the volume of the cell component usingthe data network.
 7. A computer-readable medium comprising programinstructions for performing the steps of: receiving a specification of aclass network; receiving a specification of a process hierarchy, whereinthe process hierarchy includes process steps; acquiring pixel values ofan image that includes cell components, wherein certain of the cellcomponents are marked using fluorescence in situ hybridization;performing the process steps of the process hierarchy to generate a datanetwork that includes an object, wherein the data network is generatedby linking selected pixel values to the object based on the classnetwork; and counting the marked cell components using the data network.8. The computer-readable medium of claim 7, wherein the objectcorresponds to a marked cell component.
 9. The computer-readable mediumof claim 8, wherein the marked cell component is an Her-2/neural gene.10. The computer-readable medium of claim 8, wherein the marked cellcomponent is a centromere of a chromosome.
 11. The computer-readablemedium of claim 8, wherein the marked cell component is a nucleus of acell.
 12. The computer-readable medium of claim 7, wherein one of thecell components has an area, having further program instructions forperforming the step of: determining the area of the cell component usingthe data network.
 13. A method comprising: specifying a class networkhaving a class, wherein a membership function defines a likelihood thatan object of a data network belongs to the class; specifying a processstep that is part of a process hierarchy; acquiring table data valuesthat include pixel values of an image of cell components; generating thedata network by generating the object of the data network and byselectively linking selected table data values to the object accordingto the class network and the process hierarchy; and quantifying the cellcomponents.
 14. The method of claim 13, wherein the cell components aregenes marked with a fluorescent marker, and wherein the quantifyingincludes counting the genes.
 15. The method of claim 13, wherein theimage depicts a slice from a z-stack on a slide taken from a biopsytissue, wherein the table data values include pixel values of a secondimage of cell components, and wherein the second image depicts a secondslice from the z-stack on the slide taken from the biopsy tissue. 16.The method of claim 15, wherein a first cell component in the image is adistance in the biopsy tissue apart from a second cell component in thesecond image, and wherein the quantifying includes determining thedistance in the biopsy tissue between the first cell component and thesecond cell component.
 17. The method of claim 13, wherein each of thecell components has a volume, and wherein the quantifying includesdetermining the volume of each cell component.
 18. The method of claim13, wherein the image is obtained using a confocal microscope.
 19. Themethod of claim 13, wherein the image is one of a plurality of parallelplanar scans of human tissue, wherein the generating the data networkincludes generating sub-objects by linking selected pixel values of thescans to the sub-objects, and wherein the generating the data networkincludes linking the sub-objects of adjacent scans.
 20. The method ofclaim 19, wherein the linked sub-objects of adjacent scans represent thecell components.
 21. The method of claim 13, wherein the table datavalues include items of metadata relating to the image.
 22. The methodof claim 13, wherein the table data values comprise values associatedwith a plurality of scans taken at different time intervals, wherein thegenerating the data network includes generating sub-objects by linkingselected values in each of the scans to the sub-objects, and wherein thegenerating the data network includes linking the sub-objects of scanstaken in adjacent time intervals.
 23. The method of claim 22, whereinthe linked sub-objects of scans taken in adjacent time intervals depicta movement of a cell component.
 24. The method of claim 13, wherein thetable data values comprise values from a first group of parallel planarslices of a three-dimensional data set of a cell taken at one time andfrom a second group of parallel planar slices of the three-dimensionaldata set of the cell taken at a different time, and wherein the valuesfrom the first group and the second group of parallel planar slicesdepict movement of the cell components.
 25. The method of claim 13,wherein the table data values include patient data of a patient, whereinthe patient has a gender, an age, a weight, a height, blood values,prescribed medications, a number of children, a family history ofailments, a history of breastfeeding, a history of smoking and a historyof drug use, and wherein the patient data is taken from the groupconsisting of: the gender, the age, the weight, the height, the bloodvalues, the prescribed medications, the number of children, the familyhistory of ailments, the history of breastfeeding, the history ofsmoking, the history of drug use, and tissue analysis results of thepatient.
 26. A computer-implemented network structure, comprising: adata network including a first pixel value, a second pixel value, afirst link, a second link, a first object and a second object, whereinthe first pixel value is part of a first data table and the second pixelvalue is part of a second data table, wherein the first data tableincludes pixel values from a first image slice of a biopsy tissue,wherein the second data table includes pixel values from a second imageslice of the biopsy tissue, wherein the first link links the first pixelvalue to the first object, and wherein the second link links the secondpixel value to the second object; a class network including a class,wherein a membership function determines that both the first object andthe second object belong to the class; and a process hierarchy includinga process step, wherein the process step includes a domain specificationand an algorithm, wherein the domain specification designates the class,wherein the class corresponds to cell components of the biopsy tissuethat have been marked using fluorescence in situ hybridization, andwherein the algorithm counts the cell components that belong to theclass.
 27. The network structure of claim 26, wherein the cellcomponents are Her-2/neural genes.
 28. The network structure of claim26, wherein the first image slice and the second image slice are part ofa z-stack on a slide taken from the biopsy tissue.
 29. The networkstructure of claim 26, wherein each of the cell components has a volume,and wherein the algorithm determines the volume of each of the cellcomponents.
 30. The network structure of claim 26, wherein the firstdata table includes an item of metadata that relates to the biopsytissue, and wherein the algorithm is performed based on the item ofmetadata.